Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching

Offline imitation learning (IL) promises the ability to learn performant policies from pre-collected demonstrations without interactions with environment. However, imitating behaviors fully offline typically requires numerous expert data. To tackle this issue, we study setting where have limited data and supplementary suboptimal In case, a well-known issue is distribution shift between learned policy behavior that collects Prior works mitigate by regularizing KL divergence stationary state-action distributions of policy. We argue such constraints based on exact matching can be overly conservative hamper learning, especially when imperfect highly suboptimal. resolve present RelaxDICE, which employs an asymmetrically-relaxed f-divergence for explicit support regularization. Specifically, instead driving exactly match policy, impose little penalty whenever density ratio their upper bounded constant. Note formulation leads nested min-max optimization problem, causes instability in practice. RelaxDICE addresses challenge supporting closed-form solution inner maximization problem. Extensive empirical shows our method significantly outperforms best prior IL six standard continuous control environments over 30% performance gain average, across 22 settings dataset